Method and systems of enhancing the effectiveness and success of research and development

ABSTRACT

Methods of doing business and systems for implementing those methods which improve the effectiveness and success of the research and development of technology such as pharmaceuticals, biotechnology, agrochemicals, medical technology, and genomics. The methods include the application of value and cost modeling methodologies to provide a pharmaceutical, biotechnology or genomics research and development organization with information and data which will assist it in making choices and decisions about its prospective products and the manner of research and development thereof in order to realize the greatest economic gain from commercialization of the products.

FIELD OF THE INVENTION

This invention relates generally to methods of doing business andsystems for implementing those methods which improve the effectivenessand success of the research and development (R&D) of technology such aspharmaceuticals, biotechnology, agrochemicals, medical technology, andgenomics. More particularly, the inventive methods include theapplication of value and cost modeling methodologies to provide aresearch and development organization with information and data whichwill assist it in making choices and decisions about its prospectiveproducts and the manner of research and development thereof in order torealize the greatest economic gain from commercialization of theproducts.

BACKGROUND OF THE INVENTION

The identification of effective and safe pharmaceutical, medical,agrochemical, biotech and genomics technologies is of great commercialand human importance. Currently, many potential medical innovations andpharmaceutical compounds which progress into the development phase aredetermined to be unsuitable for viable commercial use, being ineffectiveor inactive in humans or otherwise determined to be unsafe. The averagecost of a new drug approval is between $360 and $600 million for eachsuccessful launch and requires from about 12 to 15 years to achieve. SeePharma Exec., January 2000, Windhover Information Prentis Grabowski,1994 Journal of Health Economics, Vol. 13. Considering that only 3 in 10drugs achieve revenues greater than their development costs, id, theunsuccessful selection of prospective drug products is extremely costlyto the manufacturer, and ultimately to the insurer as well as to theconsumer. It is thus an ongoing objective of the medical, biotechnologyand pharmaceutical industries to find effective ways of reducing thishigh attrition rate.

There are many commercially available tools that utilize predictivemodels to eliminate unsuccessful products, such as drug compounds or thelike, before substantial time and money are invested in research anddevelopment. One such model is used to predict adsorption, distribution,metabolism and excretion (ADME) properties and toxicology profiles of adrug compound. Once determined, a predictive ADME or ADME/tox model isuseful for deciding on the particular courses of action to be taken insubsequent stages of the drug's development on the premise that drugcandidates having superior ADME properties have a greater likelihood ofclinical success. Examples of predictive ADME/tox tools are theBioPrint™ products distributed by Cerep, Inc., the VolSurf™ model byTripos, iDEA™ from LION Bioscience AG and the QikProp™ software bySchrodinger, Inc.

While predictive models are helpful in determining clinically sound drugcandidates, they only provide part of the overall picture. In particularthere is a need to plan research work so as to best balance scientificand commercial/cost considerations. Such “business” related factorsinclude the manner in which research operations are conducted, includingdetermining the number of targets to be researched at any one time,which tests to apply to compounds that may become active ingredients inproduct candidates, the sequence in which to apply these tests, thecriteria to apply when progressing compounds from one stage of researchand development to the next, and whether certain compounds should bedeveloped in parallel or serially with respect to each other. Anothergroup of factors not considered by predictive scientific property modelsincludes those dependent upon an organization's resource capacity andconstraints, e.g., the scope and number of scientific personnel needed,the amount of lab space and equipment required, and the costs associatedwith each of these.

Other business considerations that are pivotal in ensuring a drug'sultimate commercial viability are the potential demand for treatment ofa particular disease or condition, the available market size andcompetitors' activities related to treatment of the same condition ordisease state to which the subject drug is targeted. The attractivenessof a drug, and hence the available market share, will depend on factorssuch as the frequency of dosing that are originally determined by thechemical nature of the compound chosen for development of an activeingredient, but cannot be directly measured in the early stages ofresearch. Factors of this nature have to be projected from the emergingresults of testing during the R&D process, and the planned tests may bemodified in accordance with findings, possibly including the decision towork on a different active ingredient.

It is necessary to consider all of these business factors in combinationwith scientific factors to insure favorable risk-to-benefit andcost-to-profit ratios in the projects that discover, develop andcommercialize a drug or other medical or biotechnology product. Thefundamental problem is to capture and correctly apply an understandingof how early scientific measures of quality of a potential drug relateto economic measures of quality, i.e., sales performance andprofitability, in the market.

A number of business modeling approaches exist to assist management inmaking the right decisions and best choices to increase the likelihoodof the commercial success of a drug; however, they are not without theirshortcomings. Two well-known business modeling approaches used in thepharmaceutical industry are throughput modeling and discrete eventsimulation.

Throughput modeling looks at how many compounds, leads and developmentcandidates are expected to pass each stage, while discrete eventsimulation models the detail of tasks, sequence, time and contention forresources. A throughput model used in isolation is lacking in that itfails to assess how various decision criteria affect the quality of adrug compound and the yields that would be achieved in the later stagesof testing.

The discrete event simulation approach is able to deal with fluctuationsin work over time, which may be important in the later stages of R&D.See “A Systems Engineering Approach to New Product Development”, GaryBlau, CAST Communications, Vol. 20 No. 1, Summer 1997, pp 4-11. It israre for such models, if applied to earlier stages of R&D, to representin any depth the differences between individual examples of compounds orother research options, as these do not appear important from theviewpoint of scheduling a process such as screening as a materialshandling operation. However, in reality, there are various dimensions ofquality important in evaluating the potential commercial success of eachof the many molecules that may need to be made and screened beforeidentifying a development candidate, including, for example, activity,safety, transport properties and novelty. The presence or absence ofthese factors influences the value of the product and the cost and riskof downstream work. For example, molecules that show a lack ofselectivity are less valuable as products, and also more likely to failclinical trials, and lead series that enumerate only a small part of thevariety of active chemical structures are more likely to lead to lostsales due to early launch by competitors of equally, or more, attractiveproducts. In the reference cited above, many of these kinds ofdifference between options were combined into a single “degree ofdifficulty” affecting the time taken to work on a particular project. Ina subsequent development of this approach, the sequence of decisions indrug development was combined with a representation of subjectivesuccess probabilities for different projects at different stages;however, it was acknowledged that the complexity and creativity inherentto the discovery process makes it difficult to capture all theactivities in the discovery process. See “Risk Management in theDevelopment of New Products in Highly Regulated Industries”, Blau etal., Computers and Chemical Engineering, Vol. 24, pp. 659-664 (2000).

Where a decision making process involves the impact of a singlecriterion or variable on the potential success of a product, combiningcost and value parameters in a simple trade-off equation (cost-benefitanalysis) provides reliable insights. However, such is not the case withmultiple-attribute decision-making. While decision support methods doexist for analyzing multiple decision criteria, they are, too, notwithout shortcomings. In multi-attribute decision systems, the overalluser preference or “utility” for an option is determined in terms ofvalues of various attributes of the option and the preferences of theuser towards each of those attributes (i.e., the importance of thoseattributes). A preference function combining these attributes and theirvalues characterizes the structure of the preference model. See Keeney,R., Raiffa, H., Decisions with Multiple Objectives: Preferences andValue Trade-Offs, John Wiley and Sons, 1976. An alternative approach,conjoint analysis, has been used widely in market research to identifythe factors that contribute to consumer preference, through a process ofconsumer research and data fitting (seewww.populus.com/techpapers/conjoint.pdf). A drawback of both thesemethods, when applied to choice of research methods and researchoptions, is the subjective nature of elicited human judgments on theutility of technical and scientific measurements, which may not beobjectively based on business and economic metrics. For example, ascientist may, unwittingly, heavily weight a factor in the drugselection decision of which they have good knowledge (e.g., it isrelevant to their scientific specialty) even if, objectively, such afactor is likely to make only a small contribution to value. Even theseemingly objective goal for a method of “accuracy” is misleading sinceshortfalls in predictive reliability have two components, falsepositives and false negatives, and the relative importance of thesedepends on the consequences of each, which in turn depends on the costsand value of downstream activities.

In order to successfully model the entirety of the R&D process with theaim of guiding its improvement, it is necessary to track the multiplesources of potential failure for each of the many research options,e.g., the screening of hundreds of thousands of compounds, throughsequential stages of R&D where multiple criteria are used to selectcompounds, and where new research options are added through businessprocesses such as lead optimization. The successful modeling of the fullR&D process, in a way that takes account of uncertainty, the variety ofresearch options, capacity constraints, and can incorporate newfindings, has been an unsolved challenge.

Accordingly, there is still a need for comprehensive methods ofimproving the effectiveness of technology and scientific research inorder to maximize the economic value of products that come out of suchresearch, taking account of the operating costs and capacity ofresearch. More particularly, there is a need for methods which take intoconsideration both scientific and business factors in the context of aresearch and development process having multiple measurable sources ofvariety (including the differing measures for screening results and alsoboth descriptors and calculated values for structure and properties ofcompounds, series, targets and assays) and criteria (variables used asthe basis for decision making) which are applied in sequence. Whileexplicit probability distributions may be used with these approaches tomodel variation within such factors and correlations between them all, alarge amount of data is required, which would be prohibitive if allpossible combinations were included in the model. Methods are thereforerequired that can efficiently capture the essential results fromcompound screening and biological target evaluation in a way suitablefor use in guiding subsequent decisions. It would be additionallyadvantageous if such methods provided for adaptive learning in whichcertain estimates of relationships between these variables, includingvariables usable as criteria, are adjusted or re-weighted in response toscientific and business outcomes, thereby refining the processes ofproduct selection, research and development. Finally, to minimize usersubjectivity, such methods should preferably relate scientific measuresto business and economic outcomes using economic value as the commonfigure of merit.

SUMMARY OF THE INVENTION

The present invention provides new methods of doing business and systemsfor implementing these methods for assisting an R&D organization inmaking choices and decisions about its prospective products and theresearch and development thereof. The subject methods facilitate theeffective compilation and synthesis of certain information and dataavailable to the R&D organization. The R&D organization can subsequentlyuse the compiled and synthesized information and data to realize thegreatest overall economic gain from further research, development andcommercialization of the products, and can better judge what additionalinformation would best guide improved processes and research methods inthe future.

An important aspect of the invention is the consideration of bothscientific and business-related factors, and the relationships of causeand effect between them, in making decisions and choices at the variousstages in the research and development process. More specifically, inone variation, the invention comprises a method of doing businesscomprising at least the following: (a) compiling scientific and businessdata regarding product or development candidates and R&D capacity,resources and flow; (b) applying a combination of value and costmodeling methodologies to data; (c) creating, from data, priorknowledge, experience and structured experiment, a model in the form ofa probabilistic network that contains conditional probabilityrelationships, using scientific and commercial measures and estimates ofquality, between the envisaged product and the research options withinthe R&D process that are precursors to this product (e.g., a target,lead and lead series, and development candidate); and (d) makingdecisions and choices regarding research and development processes basedon results of methodology application of (b) and (c) and the use of sumsor integrals over the probabilities of contiguous states, withinvariables represented in the network that are affected by changes incriteria, to estimate, for varying thresholds on selection andenumeration criteria, fractional changes to flow and therefore, throughcumulation of such changes over successive stages of work, impacts onthe expected rate of issuing candidates and on capacity use, togetherwith quality distributions and hence candidate value projected from theselected thresholds using the probabilistic network. The decision-makingprocess of (d) may further comprise refining the data compilation of (a)and the methodology application of (b) and (c). A goal of the inventionis to commercialize highly successful and profitable products based onhighly efficient research and development processes. The steps of themethod may be carried out by the same entity or by several entities allon behalf of the entity researching, developing and/or eventuallycommercializing the product.

The inventive methods may include a combination of various techniquesand methodologies which can be applied at various stages of the researchand development process to generate information which can then be usedto make various decisions at any stage of research and development,including the planning, implementation, outcome review and improvementof research tasks and methods. Examples of such techniques andmethodologies include conventional decision analysis (e.g., decisiontrees), expert systems, statistics, neural networks, belief networks(ie., Bayesian networks), steady-state modeling, optimization methods(e.g., gradient descent, maximizing value within constraints), economictheories (e.g., marginal cost formulations), etc.

In one variation of the present invention, the subject methods involvethe application of both value and cost modeling methodologies. A user ofthe methods determines the projected market value of a product and therelationship between such value and the attributes of product quality(typically, activity, safety margin, selectivity, toxicity, dosefrequency, dose form, novelty, ease of manufacture, etc.), also referredto as a “quality profile,” using a probabilistic network or other meansfor learning of complex relationships, including statistical analysis(e.g., multiple regression). Such measures of product quality may alsobe applied for the comparison of product candidates or developmentcandidates, as “candidate quality attributes”. Expert judgments may alsobe incorporated where necessary. At the same time, the likelihood offailure at different stages of development, where each successive stageis undertaken only for candidates passing the criteria applied in allprevious stages, may be estimated and combined with the cost, or ingeneral, resource use, of each development stage, to determine anexpected, averaged cost of development and the chance of reachingmarket, using methods such as decision trees, influence diagrams or abelief network. For each value that could be taken by the candidatequality attributes, the subject methods then provide an estimate ofvalue-added and prospective risk. A probabilistic network is then formedto represent the chain of cause and effect between the various candidatequality attributes, severally or jointly, where that chain representsthe reliability of estimates of candidate quality made directly orindirectly in the form of measures on research options during thevarious stages of research, where those measures may also be useful ascriteria for selection or enumeration processes during research, suchthat the total model, including this probabilistic network, togetherwith the aforementioned model of the relationship between qualityattributes of the candidate and business outcomes, permits thecalculation of impact of changes in the way that decisions are madeduring the research and development process. The subject methods thenallow the user to assess how the manner in which research operations areconducted is likely to impact business outcomes, according to theeffects of changed research decisions on the quality profile of thedevelopment candidates.

Assuming that the research tasks such as making and testing individualcompounds, and also, for some purposes, the handling of batches ofcompounds, can be approximated as a continuum (i.e., the precise timingrelationships can be neglected), then the subject methods provideinsight into the potential constraints on capacities and bottlenecks inthe process flow thereby enabling the user to make strategic choices anddecisions about the research and development of a product that willminimize the costs of such research and development and at the same timemaximize the net value of the resulting development options created inunit time (i.e., the volume of output of development candidates in unittime times the expected value for such a candidate, where the volume isflow times time). In this continuum approximation, the capacity at eachstage is assumed to be shared amongst multiple projects, although theneeds for capacity may differ (e.g., on in any one laboratory an assayin one project may consistently require more time than in a secondproject).

The subject methods provide insight into the potential constraints oncapacities and bottlenecks in the process flow thereby enabling the userto make strategic choices and decisions about the research anddevelopment of a product that will minimize the costs of suchdevelopment and at the same time maximize the net value of the resultingdevelopment options (the volume times the expected value). It may beconvenient to assume that the continuum model is a model of asteady-state system, for example by averaging over a sufficient numberof projects. Alternatively, the flow may be considered to vary overtime, but still averaging the work over more than one task. The amountof work to be done at a stage or within a set of tasks may also bedivided by the flow, to estimate how long, on average, the work islikely to take. It is also possible to consider the best choices for asingle project, taking into account any special product or scientificrequirements in that project, by taking the total capacity available andreducing it by the expected demand on resource from other concurrentprojects. Where information is available about such characteristics fora whole set of projects within a portfolio then predictions anddecisions can be made for the whole portfolio, weighting the projects bythe number of tasks, resources used for each task, and using theproject-specific estimates of probabilities of success at differentstages of R&D in the different projects.

With the subject methods, it is also possible to inform the best choicesfor a single project, taking into account any special product orscientific requirements in that project and the total capacity availableand reducing it by the expected demand on resource from other concurrentprojects. Where information is available about such characteristics fora whole set of projects within a company's portfolio, then predictionsand decisions can be made for the whole portfolio, weighting theprojects by the number of tasks, resources used for each task, and usingthe project-specific estimates of probabilities of success at differentstages of R&D in the different projects.

The systems of the present invention include a computer readable mediumcarrying one or more software programs each having one or more sequencesof instructions for carrying out the method steps of the presentinvention. The software program may include means for receiving one ormore sequences of instructions from a user of a computer system forinitiating operation of such software program and for providing variousparameters for performing the method steps.

Information obtained by the present invention is used to enhance theprocess by which compounds such as drugs are selected as developmentcandidates, and developed up to the point of regulatory approval andsales. By enhancing the efficiency of these R&D processes the efficiencyof the overall method of doing business is enhanced. The presentinvention endeavors to decrease research and developmental costsrelative to the value of the products that emerge from R&D, therebyincreasing profits. The methodologies and systems of the invention maypossess the advantage of being an interactive, comprehensive andoutcome-responsive tool for the pharmaceutical and biotechnologyindustries.

These and other objects, advantages, and features of the invention willbecome apparent to those persons skilled in the art upon reading thedetails of the invention as more fully described below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a chart illustrating the types of tasks performed by thevarious personnel roles involved in the R&D process, and the reiterativeor learning aspect of the methods of the present invention as applied tothe R&D decision-making process.

FIG. 2 is a chart illustrating typical stages of the discovery phase ofa research and development project and the associated quality profilewhich links the research options considered in these stages withcapacity and resource use.

DEFINITIONS

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Still, certain elements aredefined below for the sake of clarity and ease of reference.

A “bottleneck,” as used herein, is a stage within a process in whichqueues are consistently observed as a result of contention between tasksfor limited resource or fixed assets (i.e., there is a criticallimitation on capacity).

A “candidate”, as used herein, is used interchangeably with developmentcandidate and product candidate, in either case, a potential product forwhich further development is required to establish viability or safety.

“Capacity,” as used herein, is the maximum rate of work (i.e., thenumber of tasks in unit time) that can be performed given a set level ofresource(s) available for performance of each task that may optionallybe shared between tasks that are performed in parallel, or more widelybetween tasks at different stages of research that are occurringconcurrently within differing projects. The availability of capacity maydepend on the timescale of analysis. With sufficient planning andexpenditure, new capacity can be obtained. “Effective capacity” may beless than theoretical capacity where there are a number of differenttypes of work (e.g., different scientific assays) which can all beperformed on the same equipment or in the some facility or by the samespecialists, as effective capacity accounts for the time taken to switchbetween one type of work and another (e.g., clean-down, set-up andcalibration activities). Effective capacity may therefore depend on thevariety of work and the way it is scheduled; frequent switching betweentypes of work (e.g., different assays for different projects within anhigh throughput screening facility), or a fluctuating flow of work, willreduce effective capacity, as defined here relative to the average flowof work, as would a pattern of research conduct comprising the executionof only a few projects at any one site where the stages within thoseprojects are lengthy and strictly sequential, and resources at that siteare not sufficient for all projects to execute the same stage inparallel, so that work in one project at a given stage would be likelyto be significantly delayed by a different project working at the samestage.

“Criterion” is a measure used in selection processes or the direction ofchange in enumeration of research options.

“Cumulative selection”, as used herein, means that there is more thanone selection step over the time sequence modeled and that the ratios offlow, determined in each selection step, are multiplied together todetermine the overall impact of cumulative selection on the flow.

“Decision boundary”, as used herein, means the condition of a criterionrequired for a change in a course of action.

“Direction of enhancement”, as used here in the context of anenumeration, means the feature (or characteristic) or combined features(or characteristics) of a research option that are used as measures whendeciding which further research options to include in an enumeration.For example, there may be a chemical compound (“lead”) that haspromising properties and a lead series is to be developed that consistsof further compounds with some similarities and some differencesrelative to the lead; then a direction of change may, for example, bedefined as adding functional groups in one or more locations of varyingsize (e.g., methyl, ethyl, propyl, butyl), or varying atomic weight(e.g., fluoro-, chloro-, bromo- iodo-) or varying electronegativity, orvarying charge, or varying strength of hydrogen bond acceptance ordonation, or many other such changes that can be ranked in somepredetermined way, as known to medicinal chemists. A variable used inQuantitative Structure-Activity Relationships (QSAR) could be such adirection of change. A direction of change may also be determined overensemble properties of the research option such as size, number ofrotatable bonds, selectivity, prevalence (for example, of receptors,enzymes or metabolic pathways) within one or more population groups, andalso where those properties arc financial or economic in nature, or arefrequencies of occurrence, such as the number of medicinal patents inwhich a particular chemical or biochemical feature occurs or where theyare estimated likelihoods, such as the chance of competition.

“Enumeration”, as used herein, is the addition of further researchoptions of the same kind as an existing research option. Selection maybe combined with an enumeration process as follows: a direction ofchange is determined, many possible new research options, including aninfinite number of new research options, are then within the range ofconsideration, and specific research options, or research options withina finite range on one or more measures, are selected for practicalevaluation.

A “false negative,” as used herein, is a result from a test thatindicates that a research option is of low quality, when in fact, oraccording to a different test deemed more reliable, that research optionis of high quality on the criterion being considered (e.g., compoundactivity as a drug against a given biological target). This kind oferror in a prediction may, if used in the decision-making process,wrongly eliminate research options that could have generated net value.The rate of false negatives as a fraction is (1-sensitivity) for a testthat measures a characteristic of positive quality (e.g., activity) and(1-specificity) for a test that measures a characteristic of negativequality (e.g., toxicity).

A “false positive,” as used herein, is a result from a test thatindicates that a research option is of high quality, when in fact, oraccording to a different test deemed more reliable, that research optionis of low quality on the criterion being considered (e.g., compoundactivity as a drug against a given biological target). This kind oferror in a prediction, if used in the decision-making process, leads tounnecessary costs and/or allocation of fixed resource capacity in theinvestigation of research options that do not have a genuine potentialto generate net value. The rate of false positives as a fraction is(1-specificity) for a test that measures a characteristic of positivequality (e.g., activity) and (1-sensitivity) for a test that measures acharacteristic of negative quality (e.g., toxicity).

The term “flow,” as used here, is the average amount of work (number oftasks) being completed in unit time. The units of flow depend on thenature of the task. In research, this is typically the number ofcompounds screened in unit time. Flow is measured in the same units ascapacity and the ratio of flow to capacity is the “loading” on capacity.The detailed measure of capacity and flow will change from stage tostage of research according to whether individual compounds or series ofcompounds are being considered, and therefore our model includesconversion factors to be supplied by the user such as the number ofchemical compounds made and tested during a campaign of high-throughputscreening. Flow is also influenced by the use of multiple cycles ofwork, each of a given batch size, in work such as lead optimization. Forexample, if in a project there are expected to be 6 cycles within thestage of lead optimization with each making and then evaluating a batchof 50 compounds, and if an organization handles 5 such projects in ayear, then the flow of compounds to be made and evaluated at this stageis 1500 (6×50×5) compounds per year.

As used herein, the term “modeling” means an abstract representation ofreality. Models allow a user to explore the likely consequences ofalternatives and guide them towards a course of action that helps tomeet their objectives. For example, in the analysis of a portfolio ofdrug development projects, the details of expected timing of work areimportant to a realistic model, as costs of payment to externalorganizations for clinical trails are high, and the work on anindividual project is very peaky over time. However, to represent theearlier stages of R&D it can be sufficient to make the assumption thatas one project finishes, use of a given resource will switch to anotherproject, so that work is approximated as a continuum, or further, as acontinuous flow. This allows the model user to look at the best way tooperate R&D across a set of projects.

As used herein, “an ordered classification”, also abbreviated as “anordering” is the result of application of one or more measures toresearch options such that they are ranked in preference. Thatpreference may then be used to influence the future conduct of theresearch, for example by selection only of some options for furtherinvestigation at any one time. The choice of research options may bewithin either a selection or an enumeration process. The scale used maybe continuous or discontinuous (discrete). In the case where theinvention uses a belief net as the probabilistic network (definedbelow), a continuous scale is discretized at equal or unequal intervals,including possibly transformations of the measure (such as takinglogarithms). Where unequal intervals are used these are best chosen sothat the intervals are more closely spaced around values important tothe user when considering alternative decision thresholds, allowing forselections that are likely already to have been made. For example, amodel of hit confirmation will preferentially sample properties for hitsnot for all molecules originally screened. An ordered classification mayinclude a classification using multiple measures where those measuresare numerically combined in a linear or nonlinear coordinatetransformation, in which case each of the new coordinates is availableas a potential measure for ordering, and classification may use severalor all of these measures. In a preferred embodiment, each of thesetransformed coordinates is presented to the user as a separate choice,where later choices are made conditional on the selections made.

As used herein, “quality” when applied to a “product” refers to thatproduct's fitness for a particular purpose, wherein a product may be anartifact, e.g., an active ingredient (compound), of a final product,e.g., a drug formulation. Quality is preferably defined over a number ofattributes that may be partially interdependent. The final quality of adrug, for example, is measurable in terms relevant to patient treatment,sales value and cost of production, such as the therapeutic margin(therapeutic index) in humans, safety, convenience of dosing, stabilityand complexity of manufacture of the formulation. A drug of higherquality will have a higher expected net present value taking intoaccount sales, profit and loss over a range of possible futures. At anearlier stage of research, these attributes of drug quality may notdirectly measurable, as evidence is available only from calculationsbased on chemical structure, or from physicochemical measurements,in-vitro and in-vivo (animal) tests, bioinformatics, proteomics andgenomics. Empirical results and scientific theories often indicate acausal relationship between research findings on a compound and itsquality if developed as a drug. For example, if a compound is detectedas a mutagen for a bacterium in an AMES test, it is likely to becarcinogenic if given to humans. A compound that binds only weakly to abiological target, such as an enzyme or receptor, in a high-throughputassay, will not turn out to be an active drug where this target is partof the necessary mechanism for intervention in the disease. Therefore,the definition of “quality” with respect to early-stage research is thecalculated or measured values for compounds, targets and chemical serieswhich tend to correlate with later drug quality. These values can be foran individual compound, for a series of compounds with some commondescriptor, or for the biological target (e.g., the percentage ofpotential patients in which an drug intervention based on this targetwill significantly reduce disease symptoms, or effect a total cure).Such values may typically be used as criteria for selection orenumeration during the research process. The term quality may also beused to describe the reliability of an assay, which is a test for theinteraction between a compound and a sample of a target, and byextension may also be used to describe the fitness for purpose withinresearch of use of an animal, organ, cellular or other physical researchmodel or technical mechanism upon which such an assay relies.

As used here, a “probabilistic network” is a belief net, neural network,system of logic or other means for the representation of conditionalprobabilities in which a change of assumption or findings about onevariable may be rapidly propagated to determine and report theprobabilities of other variables. Where algorithms are known to thoseversed in the art of statistics, or artificial intelligence, such thatthe conditional probabilities and/or structure of a probabilisticnetwork may be estimated from individual examples of findings, withoutthe need for explicit knowledge of probabilities, then this is termedhere a “trainable network”. A belief network, otherwise known as aBayesian network or belief net, assists statistical inferences of theprobability of the occurrence of an event or state given knowledge aboutone or more other events or states. This probability of occurrence isknown to statisticians as a “marginal probability” when calculatedsumming exhaustively over possibilities for other events or states and a“contingent probability” when calculated after selection of limitedpossibilities for other events or states, and both are described hereinas a “local probability”. A neural network of appropriate structure(e.g., back-propagation network) also assists statistical inferences butthese are not explicitly localized within the network, althoughsystematic changes observed within such a network may still beinterpreted in terms of probability relationships. Both kinds of networkare depicted in terms of an acyclic graph of nodes (vertices) connectedby links (edges) in which each node stands for a variable of interestand each link is used to propagate changes in value from one node to thenext, combined in a way that is determined by properties of thereceiving node. In a belief net, each link stands for a statisticalrelationship between the pair of variables at the nodes which it joins,which may include a causal influence from antecedent node (e.g., X) todescendant node (e.g., Y), expressed here as X→Y. If two nodes X and Yhave no direct path between them (they are ‘disconnected’), there is nodirect statistical relationship between the variables, although theremay be indirect (but conditionally independent) relationships as aresult of links to common nodes, (e.g., Z→X and Z→Y). In anyprobabilistic network used within this invention, for each variable,possible states are defined, which for a continuous variable arediscrete sample intervals chosen to provide a balance between accuracyand speed for computation. From a functional relationship based onscientific theories, scientific findings or expert judgment, theconditional probability relationship between directly connected nodes isexpressed, for each combination of states of direct antecedents, as atabulation of probabilities over all states of the descendent node. Theoverall or joint probability distribution function that relates all thestates of all the variables is then uniquely defined. Probabilisticnetworks are useful to provide a convenient means for expressingassumptions about statistical relationships; they facilitate economicalrepresentation of joint probability functions, and facilitate efficientinferences from observations. The existing state of the art allows,where the network is a trainable network, the most likely probabilisticnetwork to be estimated through a training process using sets ofempirical observations even where there are unobserved variables (called“hidden nodes”), so that belief nets are trainable networks. SeeCausality: Models, Reasoning and Inference, Judea Pearl, CambridgeUniversity Press, 2000.

The term “reliability,” as used herein with respect to a measurementmethod or calculation is the extent to which that method or calculationcorrectly predicts outcomes that can be independently measured.

As used herein, “research options” are the subject matter for choices oncontent for R&D, including targets, assays, compounds, lead series,chemical features, and compounds that have survived previous selectiondecisions (e.g., leads, development candidates, product candidate). Theproduct is a research option that finally passes all hurdles indevelopment. Such research options exclude the therapy area within whichresearch is being conducted, which acts as a context for the research,and also exclude the choice of conduct of research (such as thesequencing of tasks, the method of choice of research options, and thespecific criteria used for selection of research options (see definitionof criterion), and the levels of thresholds applied to criteria), withthe exception that an assay is included as a research option in thecircumstances where assays may be selected by applying measures of assayquality (e.g., false positive fraction, false negative fraction) ascriteria. Elsewhere in a different or even the same research model, anassay may be (additionally) represented as a node (variable) in aprobabilistic network providing one or more criteria through which thequality of other research options may be estimated.

As used herein, the term “resource” is something needed for a task to beperformed, for example, equipment, personnel or materials (e.g., acompound for screening).

As used herein, the term “risk” in the context of an R&D process isexpressed as a distribution over possible outcomes of R&D projects. Ifan economic measure is applied to each of these outcomes, standardstatistical measures (e.g., standard deviation) are used to assess“economic risk.” This distribution can be modeled through analysis ofthe sequence of stages and the chance of passing each stage. This chancecan be decomposed into the chance of progression of individual compounds(compound attrition) and the chance of the project failing for otherreasons, “independent attrition”, which may be due to problemsassociated with correctly understanding the biology of the disease andtarget, or associated with commercial difficulties (e.g., a competitor'spatent covers the chemical family being tested).

“Selection”, as used herein, is a process of choice amongst researchoptions, possibly including more than one measure, including thedefinition of a range on a continuous scale in one or more variables,for selection amongst an infinite number of research options. It canmean either a final decision to progress some research options andreject others, or a prioritization decision between research options. Itcan also mean a decision to process different research options indifferent ways, for example applying different subsequent criteria.Selection can also take place in two or more successive steps ofapplication of criteria where research options are marked in the firststep as having provisional properties and then in the second selectionstep, further criteria are applied and the results are combined todetermine which research options are preferred overall. Selection mayalso be applied in the form of a ranking where preferred researchoptions are progressed first in sequence and, according to subsequentfindings for such options, further research options are considered asneeded until the project or stage succeeds or is abandoned or restarted.

A “site,” as used herein, is a location of work where tasks of the samekind are likely to share resources. A single site could be distributedover a geographical region if test samples or resources (typicallypeople) are considered to be mobile.

A “stage” of research and development, as used herein, consists of theactivities required in order to move a drug target, assay, compound,lead series, lead or development candidate, or other such options withinbiotechnology industry, from consideration to selection. The R&D stagesare generally sequential for a given project but may overlap in timeand, in some instances, require repetition prior to progressing to thenext stage. More particularly, once a stage of work has completed,either there are enough successful outcomes to start the next stage insequence, or it may be necessary to recycle and return to the start ofthe same, or an even earlier stage, with a new family of compounds fortest, with new or improved assays, or even with a new biological targetfor investigation. Different stages of different projects will typicallybe performed at the same time, such that specialized resources are usedefficiently and without long idle periods.

As used herein, a “task” is used interchangeably with “activity” to meana unit of work, for example the application of a test to a compound orbatch of compounds.

As used herein, the term “test” refers to a research method oftenperformed with automated equipment, or used to describe a computationalprocedure such as the calculation of chemical properties, in eithercase, for the purpose of guiding selection of a research option oroptions. When applied to a set of compounds, a test yields a “testresult” for each compound

As used herein, a “therapy area” or “therapeutic area,” is anorganizational unit of an R&D group in which one or more biologicaltargets for intervention in the disease process are being considered.Typically a therapy area is focused on the treatment of a number ofdiseases which are commonly related based either on the part of the bodywhich they affect (e.g., the central nervous system), on the type ofdisease agent they involve (e.g., antibacterial), or on the type ofdisease process they involve (e.g., anti-inflammatory). A therapy areatypically includes several projects considering multiple modes of actionof drugs on targets and/or the potential spread of disease treatmentsavailable through a single mode of action (a mode of action is the wayin which the drug interacts within the patient with large molecules suchas proteins to create a useful therapeutic effect).

As used herein, a “threshold level” for a test is the numerical cutoffvalue (for continuous variables) or (for categorical variables) thecategory boundary between passing and failing compounds during screeningor, more generally, between passing and failing research options (e.g.,potential biological targets for research or, where the research stageis choosing an assay for use with the target, assays suitable for use).In some tests, both high and low values indicate poor (or, conceivably,good) quality, requiring the use of two threshold values defining anintermediate range there between. For a binary result (categories are“true”/“false”), there are three threshold levels and four possibleapplications of these: pass all research options, pass none of theresearch options, pass on a “true” result, or pass on a “false” result.

The term “value” as used herein with respect to a research option meansthe expected present value, the risk-adjusted and discounted stream offuture post-tax earnings. “Net value” is the gross value less the costsincurred.

As used herein, “work” is the application of one or more resources toeffect completion of one or more tasks.

A “computer-based system” or “computer system” refers to the hardwaremeans, software means, and data storage means used to analyze theinformation of the present invention. The minimum hardware of thecomputer-based systems of the present invention comprises a centralprocessing unit (CPU), input means, output means, and data storagemeans. The computer systems may further include a display means and userinterface means (e.g., keyboard). A skilled artisan can readilyappreciate that any one of the currently available computer-based systemare suitable for use in the present invention. The data storage meansmay comprise any manufacture comprising a recording of the presentinformation as described above, or a memory access means that can accesssuch a manufacture.

A “processor” references any hardware and/or software combination whichwill perform the functions required of it. For example, any processorherein may be a programmable digital microprocessor such as available inthe form of an electronic controller, mainframe, server or personalcomputer (desktop or portable). Where the processor is programmable,suitable programming can be communicated from a remote location to theprocessor, or previously saved in a computer program product (such as aportable or fixed computer readable storage medium, whether magnetic,optical or solid state device based). For example, a magnetic medium oroptical disk may carry the programming, and can be read by a suitablereader communicating with each processor at its corresponding station.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before the present invention is described, it is to be understood thatthis invention is not limited to specific method steps described, assuch may, of course, vary. It is also to be understood that theterminology used herein is for the purpose of describing particularembodiments only, and is not intended to be limiting, since the scope ofthe present invention will be limited only by the appended claims.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although any methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of the present invention, the preferred methodsand materials are now described. All publications mentioned herein areincorporated herein by reference to disclose and describe the methodsand/or materials in connection with which the publications are cited.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “and”, and “the” include plural referents unless thecontext clearly dictates otherwise. Thus, for example, reference to “adrug” may include a plurality of drugs and reference to “the step” mayinclude reference to one or more steps and equivalents thereof known tothose skilled in the art, and so forth. Further, the term, “a set”includes a set containing only one item.

The publications discussed herein are provided solely for theirdisclosure prior to the filing date of the present application. Nothingherein is to be construed as an admission that the present invention isnot entitled to antedate such publication by virtue of prior invention.Further, the dates of publication provided might be different from theactual publication dates which may need to be independently confirmed.

The present invention provides new methods of doing business and systemsfor implementing those methods whereby a technology, biotechnology,pharmaceutical or genomics organization or the like is assisted inmaking choices and decisions about its prospective products and theresearch and development thereof in order to realize the greatesteconomic gain from commercialization of the products.

An important aspect of the invention is the consideration of bothscientific and business-related factors and the cause and effect betweenthem in making decisions and choices at the various stages in theresearch and development process. The invention provides a way ofmodelling and facilitating a continuously reiterative, closed loopprocess by which the various decision makers can redefine each task asneeded at each stage of the research and development process based onthe information gleaned from implementation of the subject methods, fromthe R&D process itself and from wider industry or public domaininformation as it becomes available.

FIG. 1 provides a chart that illustrates this reiterative aspect of thepresent invention in the context of some of the tasks to be accomplishedby the various roles within the overall R&D process. Generally, there isa hierarchy of these tasks which first involves defining the decisionmaking process, i.e., deciding which tests to perform and the thresholdlevels for each test, and then applying the decision-making process to aselection of specific research options for progression (e.g., targets,series, compounds, etc.). For purposes of describing the invention, thevarious high-level tasks have been categorized as relating to (1)setting goals and analyzing risks and tradeoffs; (2) planning andexecuting the R&D process; and (3) reviewing outcomes and making newplans to improve the process based on those outcomes; however, suchcategorization is intended to be exemplary and it is recognized thatother types of decisions and tasks exist and are not intended to beexcluded herein.

Each of these types of tasks are made or performed either across thewhole R&D process or for a segment of the process, comprising one ormore sequential stages. Where these stages exclude the final launch andsales of a product, then the outcomes from these stages are valued bytaking into account the range of potential outcomes, including the costsand likelihood of success of all the R&D stages that remain to beperformed before the product launches.

The potential users of the present invention have been looselycharacterized within three broad roles, performing “strategic planning,”“process implementation,” and “technology evaluation and changeinitiatives;” however, such categorization is intended to be exemplaryand it is recognized that other roles may exist in a particular R&Dorganization, and that roles may be shared by individuals or teams. Inparticular, implementation of a research process may be delegated to theproject manager or project team with discretion to perform the decisionsand tasks described.

The arrows on the left-hand border of the chart of FIG. 1 indicate thereiterative or learning aspect of the present invention. Morespecifically, based on a review and analysis of the outcomes (task type3, which includes specific tasks (c), (f) and (i)) in the different userroles), the goal setting and risk analysis (task type 1, which includesspecific tasks (a), (d) and (g)) and the planning and execution of theprocess (task type 2, which includes specific tasks (b), (e) and (h)) toachieve the set goals are adjusted or recalibrated in order to optimizethe next set of outcomes. This learning loop is now described for eachof the strategic planning, process implementation, and technologyevaluation/research improvement roles.

Strategic Planning

Task (a), i.e., the setting of goals and the analysis of risks andtradeoff within the strategic planning of R&D, involves assessing therisks and tradeoffs between the quality and the quantity of anticipatedresearch output, to set success measures that are likely to apply tomore than one research project. Such output consists, for any stage, ofthe research options put forward for further development or selection inthe following stage, culminating in adding value at the end of R&D viaproduct registration, or possibly at an intermediate stage via licensingintellectual property to another organization. This decision/taskrequires knowledge of industry-wide and company-specific data and mayinvolve the use of conventional statistical, financial and managementscience methodologies, such as multiple regression, or factor analysis,decision tree analysis, time discounting, and calculation of a riskpremium on the cost of capital based on any undiversified risks.

Task (b), i.e., the planning and execution of the best process to meetthe goals identified in (a), involves identifying potential bottlenecksin the R&D process and allocating resources to the investigation ordevelopment of particular products, and the compounds or series that maybe developable into each product, at the cost of the attrition of otherproducts or potential products. It is necessary to decide how steep acut to make amongst the available research options at each stage, insuch a way as to avoid overloading the expected capacity. Thesedecisions/tasks may be facilitated based on the use of steady-statemodeling and by the iterative calculation of capacity loading as aresult of changes to the standard sequence of the research process thatwould, without new evidence, be applied to all new projects.

Task (c), i.e., the review of outcomes of previous tests in one or moreprojects, preparatory to the identification and then implementation ofimprovements to the R&D process in view of those outcomes, involvesreviewing the successes and failures of previous company and possiblywider industry projects, and identifying the causes of each so thatsuccess can be repeated and failures can be avoided moving forward,within the strategic planning role by adjusting the goals, cross-projectpriorities, and total resources available for work. This task may befacilitated based on at least the use of probabilistic networks to learnstatistical correlations, and in particular, causal relationships. Theinformation and data learned based on this decision/task is then used tofurther refine and calibrate decision/tasks (a) and (b).

Process Implementation

Task (d), i.e., the setting of goals and the analysis of risks andtradeoff at the process implementation stage of an R&D project, involvesdeciding on the scientific tests that should be employed and whether toemploy them in sequence or on a parallel basis. The decision on whetherto do work in parallel involves balancing the cost of utilizing manyresources at once, and the resulting impact on other projects, againstthe advantages of faster progress on a given project and of combining orweighting multiple numerical results over the various tests beforedeciding which research options to progress. This form of numericalweighting, which is a standard practice for multi-attribute decisionmaking in R&D process, allows for a better balance in quality overmultiple criteria or test results A, B, C etc, as a very favorableresult C can be set against slightly adverse results A and/or B. In thesequential use of tests, on the other hand, an research option mayalready have been eliminated on test results A or B alone and so thefinding C would not be available for consideration over all the originalresearch options. Task (d) may involve the combination of variousmethodologies of the present invention. For example, the conditionalrelationships between test results such as A, B and C, and later projectoutcomes, may be assimilated within a probabilistic network (e.g.,Bayesian network/belief net). The capacity loading calculations are thenmade according to which tests are used and to the sequence of testing ina way that is iterative with the setting of thresholds, includinggeneralized thresholds for multiple variables as defined above, so asbetter to understand the range of possibilities and benefits ordisadvantages for parallel testing and numerically combining results.

Implementation of task (e), i.e., the planning and execution of the R&Dprocess at the process implementation stage of an R&D project, assumesthat the tests to be used and their sequence or parallel nature havealready been decided in task (d). It often times involves choosing thethreshold levels for the various scientific tests determined above whiletaking into consideration predictive reliability, the impact of falsepositives and false negatives, and certain other project-specificfactors such as the anticipated chance of failure in subsequent testing,which if high, reduces the opportunity cost of a false negative. Thistask may be facilitated based on the use of modeling, notably theinteractive calculation of loading on capacity on the current andsubsequent stages according to the thresholds set in the current stage,using prior knowledge (such as from task (f)), stored permanently ortemporarily in a belief net), about relationships between qualitycharacteristics, to carry out the process of inference of the proportionof compounds passing or failing later R&D tests, and the change in netvalue of the resulting research options, related to the use of capacityand potential marginal costs of capacity additions.

Task (f), ie., the review of outcomes of running the R&D process todate, preparatory to improvements to the R&D process in view of thoseoutcomes, involves assimilating current project findings into the priorcompany and industry findings from task (c), and refining the scientificand empirical bases for the R&D process, in light of the current andcollective outcomes. This task may be facilitated based on at least theuse of probabilistic networks. The information and data learned based onthis task is then used to further refine and calibrate tasks (d) and (e)and also may contribute to the broader cross-project learning indecision/task (c).

Technology Evaluation/Change Initiative

Task (g), i.e., the setting of goals and the analysis of risks andtradeoff when considering alternatives for investment in the R&D processitself and its supporting technology (e.g., new/improved screeningequipment, new expertise, or new or enhanced software for estimatingmolecular properties), involves considering whether changes should bemade to the scientific methods employed in order to improve the process.Where the supporting technology to be considered is a predictivetechnology which provides better estimates for decision making, thenassessment of the process is accomplished through insertion, within abelief network that represents the existing research process, of nodesand links—representing the variables to be measured or predicted and theexpected reliability of prediction of the new technology—as an addition,or substituting for one or more of the existing tests. Where thetechnology is an automation technology which is used to improve the rateof existing work, then the potential value is estimated by setting thevalue of additional capacity against the marginal cost of the newtechnology, with the advantage that it is possible to compare thisimpact against alternative ways of changing research such as, instead,reducing the loading on capacity by using predictions of quality inearlier stages (known in the industry as a strategy of “earlyattrition”).

In the context of task (g), it is important to choose the best conductof research amongst all feasible alternatives, not just to estimatewhether introducing a new technology would give an improvement relativeto the current process. This task may be further facilitated by theapplication of economic theories, such as marginal cost formulations.

Task (h), i.e., the planning and execution of implementing a newtechnology or research methodology for potential use in multipleprojects, takes into consideration the potential impact on bottlenecks,quality and value in a way that allows specific advice on implementationof this new approach. As with task (g), it is important to keep in mindthat a particular technology may not be suitable for all projects or alltherapeutic areas. For example where the expected value of the outputfrom research in the project (the product candidate) is very high, thena predictive technology with a high fraction of false negatives may notbe recommended. The subject methods provide a sensitivity analysis ofnet value added by a change in research process, or the use of a newtechnology, over parameters specific to an R&D project or therapeuticarea. Within a belief net, the project portfolio is represented as achance node where each project is a discrete state, with links to othervariables that vary between projects. To select a single, individualproject from the entire portfolio, the probability of that project isset at 100%. This allows an immediate comparison of the results ofmodelling for. that project with the results of an analysis averagedover the portfolio of all projects.

Task (i), i.e., the review of outcomes from previous investments andchanges in the research process and methods, involves evaluating therobustness of the current plans for change and improvement as determinedfrom task (g), considering sensitivity to assumptions and identifyingneeds for new information. Both of these tasks may be facilitated by theuse of Bayesian networks. The sensitivity to assumptions is estimated byselecting a variable in the network and determining how the net value,the likelihood of success at any stage of R&D, or other variables ofinterest, depend on this variable. Needs for new information can beidentified through standard statistical measures of confidence, providedthe technology used for implementation of the belief net incorporates arecord of the amount of evidence used to underpin the current set ofconditional probabilities. Commonly available belief net implementationssuch as Netica™ from Norsys Corp, Canada and Hugin™ from Hugin ExpertA/S, Denmark provide facilities to identify how conclusions would beaffected by new evidence, and the existing state of the art in beliefnetworks points to ways of planning the investigation that will bestincrease the confidence in the overall reliability of inference (see,e.g., Tong, S., Koller, D., Active Learning for Structure in BayesianNetworks). From implementing the belief net, it is possible to identifythe weakest links in the factors considered in the proposition that achange should be made to the research process or that an investmentshould be made in a new technology. This kind of robustness assessmentis useful to a proponent of a research improvement initiative, as wellas to those charged with making recommendations with respect to theintroduction of a new technology, helping them to explore alternativessystematically. It is also useful in the context of task (g) inpreparing a more robust case for change or in rejecting the proposedchange. The information and data learned is then used to further refineand calibrate tasks (g) and (h).

The subject methods involve the application of both value and costmodeling methodologies, where the cost may include a use of a share offixed capacity, which results in an increase of the long-run cost ofwork through the need to replace that capacity. One means for valueassessment involves the use of probabilistic networks such as Bayesiannetworks. In the context of the present invention, the Bayesian networktracks the likelihood relationships between quality characteristics,i.e., all relevant scientific or estimated commercial (e.g., innovative)properties of a research option, product or drug and the businessoutcomes of a resulting product. The cost assessment component may bebased on the assumptions that each R&D stage through which a product isidentified and selected for development, or each set of tasks within astage if a stage is split into sequential activities, represents acontinuum, and that the R&D tasks performed on each research option arehomogenous across the research options within the organization, site,therapeutic area, or one or more R&D projects, according to the scope ofmodeling (e.g., screening one compound takes the same effort and time asscreening any other compound). It may also be assumed in the firstinstance that the personnel, space, equipment and test subjectcapacities and the costs associated with each are constant over time.Alternatively a wider analysis may be performed which studies thepotential impact of varying these parameters over time or varying theflow, including adjusting the rate of input to the first stage, forexample assuming a linear or exponential increase in the size of thelibrary of compounds available to be screened. In such a case the modelhas to take account of the average time taken for each stage of work,which is the amount of work divided by the (flow) rate of work.

Modeling a repetitive research process requires defining the flowthrough the R&D process at each stage. As mentioned above, the flowcomprises the various research options that are being considered forselection within the R&D process. The various research options includethe biological targets or compounds to be included in a real chemicallibrary for screening or a virtual library for computationalassessments, so-called “hits” which are compounds detected in a bindingassay, series of structurally similar compounds, and further compoundsmade through enumeration of each such series in the process of leadoptimization. The flow must be balanced against the process' inherentcapacities and resources and is therefore adjusted as necessary withinthe model by the addition of tests that can select out research options,allowing for the capacity use of each such test, which will depend onits specific nature and complexity, and on any necessary preparatorywork. The volume of flow is further adjusted by setting a thresholdlevel or value for one or more test parameters including calculated andderived parameters. Such test parameters are dictated by the user, andinclude both technical/scientific and business measures where available.They include but are not limited to activity level, permeability,calculated likelihood of toxicity, and molecular properties found in theindustry or company to have preferred ranges predictive of quality as adrug (such as molecular weight, lipophilicity, the number of hydrogenbond acceptors or donors, the polar surface area, the number ofrotatable bonds, or other variables derived from molecular structure).

Combining the value and cost computations of the present inventioninvolves the mathematical integration or summation of probabilities ofoccurrence of different outcomes (states) for a node in a probabilisticnetwork. This node is representative of the probability distribution(s)of the occurrence of outcomes of test results or calculations which willbe the basis for selection decisions, given the research options alreadychosen for a given research project within the planned research process,methods and technologies. The integration or summation is used toestimate the fraction of research options selected at each stage, andtherefore the change in flow, as a result of an imposition of acriterion or a change in a threshold for such a criterion.

This calculation may include choices based on previous decisions for agiven research option, i.e., a compound may not be rejected immediatelybut flagged as suspect and then rejected later more readily than acompound which has not been flagged. Also the calculation of work mayallow for alternative tests or test sequences being applied according toprevious test results.

Concurrently with estimating the fractional change in flow throughrejecting research options, the probability of occurrence within thebelief net is set to zero for all those values or discrete states thatwould be rejected at the (generalized) test threshold under assessment(where generalizing allows for the use of multiple variables in a givenchoice and the combination of variables into new sets). The belief netis then used to perform an inference process to estimate the quality ofthe outcomes at all later stages of research, and thus the resultingeconomic value and, if required, the risk, of the estimated productquality profile. These estimations are conditional on the suggestedtests and associated threshold levels.

The flows at each stage, as modified by fractional changes, and by anyindependent attrition, are set against the various capacity utilizationand resource requirements required to implement the research process forthe given selection and sequencing of tests, and levels of thresholds.The volume of research options to be assessed at each stage, or in anyparallel set of tests that may be performed separately within a stage,is compared dynamically with the available capacity to estimate themaximum feasible throughput reaching the last stage of R&D that isincluded within the analysis. Operating costs, which may depend on theflow (volume) of work, are taken into account. The flow from the laststage of R&D within the analysis is combined with the estimated value,cost and risk of further development using the quality distribution ofthe candidate product. This step of multiplying final throughput (flowof candidates as an output from the last stage) by the specific netvalue for each candidate, averaged over all the relevant parameters ofcandidate quality reported by the probabilistic network holding thepreselected research criteria, gives an estimate of the rate at whichthe suggested research process can add value.

It is also possible using a Poisson function to derive, from theestimated flow out of a stage, optionally divided into a number of sitesor a number of projects (which may give a number with a fractionalpart), the discrete probability distribution representing theprobability in any given time period of having a given number ofsuccesses. This is an important practical metric for management but maynot always lead to the same conclusions as maximizing the expected valueadded, in which case the availability of both predicted success counts(with distribution over such counts) and value creation predictions isadvantageous.

As mentioned above, the probabilistic network is used to track theconditional probability relationships amongst scientific estimates ofresearch option quality and between these and business estimates ofcandidate quality: descriptors of the product, meaningful and importantto the prescribing physician, which take part in the estimation ofeconomic value. The estimates of local and conditional probabilities maybe determined by either or both empirical data and values or scoresbased on quantitative judgments on cause and effect. The scientificproperties may be initially determined by the use of one or morepredictive models, e.g., ADME model, known by those skilled in the art.The relationship between the scientific and business qualities arerepresented as conditional probability distributions. Marginalprobability distributions may also be used to average out variables thatare temporarily of less interest, a process known within belief netapplications as “node absorption.”

FIG. 2 illustrates the general principles discussed above. There aretypically a number of stages of research and development which culminatein the provision of a product or development candidate. In each stage,various research options are considered. In early stages the researchoptions consist of targets and assays, and libraries of compounds to bescreened in these assays (i.e., high throughput screening). As a resultof this screening, hits are identified and confirmed. These hits arethen clustered into various series, and from these series, through aprocess of“lead optimization,” new compounds are synthesized. With aproject, at any one time no more than one such compound (the “lead”) maybe selected as a development candidate, possibly with other “backup”compounds to follow if development of the selected compound fails.

The quality for the research options at all stages, calculated ormeasured as test results, has a random component but is partlypredictive of factors important for final product value, with increasingreliability in later stages. The quality of a target affects the productvalue over all compounds that might be developed against that target.The quality of an assay has an indirect impact on project value byimproving the detectability of potential drugs and also the reliabilityof predictions made about them. For each stage there is a capacityconstraint, on which the fractional loading depends on the criteriaapplied, which may be detailed in terms of the resource demands of eachtest or calculation, or the supporting chemical synthesis. Thisconstraint can be elevated at a marginal cost.

Various calculations may then be made to determine the effect thatchanges to the various test parameters, or their sequencing, have on thevalue added by the R&D process. For example, the value added by a changemay be represented by the following equation:ΔNV=(ΔOR×(MV−EC))−MC−ΔOCwhere ΔNV is the change in the net value added in unit time by researchbetween alternative policies to be compared, where in the caseconsidered, the change is mainly to flow and not to quality; ΔOR is thechange in output flow rate from the last stage, here assumed to createdevelopment candidates; MV is the market value over the quality profileof the development candidates; EC is the expected cost of developmentfrom candidate up to product allowing for the set quality profile; MCis, for long-term decisions such as technology investments or changes inrecommended conduct of research, the cost of any change to capacity,amortized over the planned life of the investment using a cost ofcapital that excludes any premium for diversifiable risk, oralternatively, for short-term changes (such as individual projectdecisions), the marginal cost of the change as a cash sum, divided bythe number of time units in the period considered; and ΔOC is the changein operating cost per unit time associated with the R&D process segmentincluding costs pertaining to any change in input flow to the firststage of that segment, for example costs of access to additional targetsor biotechnology inventions.

If the change involves both flow and quality then the equation becomes:ΔNV=(OR2×(MV2−EC2))−(OR1×(MV1−EC1))−MC−ΔOCwhere the suffixes 1,2 represent the values for the two alternativesbefore and after the change, and OR, MV and EC are again output flowrate, market value and expected cost of development as defined in moredetail above.

The analysis of changes, when a test, activity or event is eitherincluded in or excluded from the research process, in conjunction withan estimate of the cost of capital, allows a user to analyze whether atest, activity or event in the R&D process is value-adding,value-diluting, value-destroying or inconsequential. By varying thethreshold level for a given test as well as the use and sequence oftests in different permutations, and the possibilities for changes tothe level and allocation of resources, it is possible to use standardoptimization approaches, such as gradient descent, to predict the bestcourse of action either by default, for all projects, therapeutic areasand/or all sites, or, by appropriate selection of resource needs,development candidate valuations, criteria modeled within the beliefnet, rates of independent attrition, and conditional probabilityrelationships, for (a) selected project(s), therapeutic area(s) andsite(s).

This optimization may be performed in up to three nested loops (i.e.,the innermost loop, the middle loop and the outermost loop) according tothe range of R&D methods that the user wishes to explore. The innermostloop involves the variation of thresholds in one or more tests in one ormore stages, up to but not including the extreme cases where either atest is not passing any research options or is passing all researchoptions (which cases belong in the middle loop). The middle loopinvolves the addition or removal of tests from a parallel set or stageof work, with corresponding changes to resource use and including, ifweighted scoring is being considered, the inclusion of new possibilitiesfor generalized thresholds within the parameters modeled in the beliefnetwork and available for use in the calculation of flow volumes. Theoutermost loop involves the exploration of different levels of capacityconstraint, and different allocations of shared resources (such aschemists) to help change the effective capacities in more than one stageof work, or parallel set of tests within a stage; for example, movingchemists from synthesizing a large library for high-throughputscreening, to making new examples of lead series in the leadoptimization stage of R&D. Alternative algorithms that give similarresults may be employed, as known to those versed in the art ofoptimization.

In one variation, the present invention provides a method of improvingthe effectiveness of an R&D process within an organization where thatorganization may include one or more sites, therapy areas and projects,and the R&D process includes a plurality of stages of selection and/orenumeration of research options and identification of one or morecandidates. Each stage comprises one or more sets of parallel tasks thatare optionally reiterated in a number of cycles. Each task comprises oneor more subtasks, wherein a batch is a collection of the one or moresubtasks. Each set of parallel tasks has an input flow and an outputflow of research options, wherein, at a final stage, the researchoptions are candidates. In particular, the method includes thefollowing: identifying quality attributes of the candidates; defining ameasure of utility over a quality profile of identified qualityattributes, wherein a combination of utility and the flow of candidatesquantifies improvements to the R&D process; identifying the types ofresearch options for each of the plurality of stages; identifyingcriteria relevant for use in selecting and/or enumerating amongstresearch options of each identified type; identifying causalrelationships amongst the criteria and between the criteria and thequality attributes; forming a probabilistic network representing thecausal relationships as links holding conditional probabilities andrepresenting the criteria and quality attributes as nodes, each nodehaving a plurality of states permitting an ordering amongst the researchoptions consistent with thresholds applicable for selecting and/orenumerating the research options, and each node holding a probabilitydistribution over the plurality of states; estimating an effectivecapacity for each stage; setting local probabilities at nodes thatrepresent prior knowledge of the research options and the R&D process;identifying alternatives for operating the R&D process; intervening inthe probabilistic network to change the probabilities of the states torepresent differences between the alternatives wherein the differencescomprise at least one of (i) the criteria to be applied, (ii) the stagein which a criterion is applied, and (iii) the threshold at which acriterion is applied; and evaluating the degree of improvement bycalculating (i) the quality profile of the candidates, (ii) the changein an average of the utility for each candidate, and (iii) the outputflow of the candidates, limited at the final stage and also at eachprevious stage by the input flow to that stage relative to the effectivecapacity. Various steps of the method may be reiterated as needed tooptimize the outcomes. In particular, the steps of setting localprobabilities through to evaluating the degree of improvements, and thesteps in between, may be reiterated as needed.

Within the context of the above-described method variation, utility is ameasure of gross economic value where the net economic value is thegross economic value less operating costs estimated from at least theinput flow into at least one stage of the R&D process. Further, theeffective capacity is considered to have a cost associated with it wherethat cost is a capital cost As such, the method may include theadditional step of comparing the net economic value with the cost of theeffective capacity.

In one variation of the above-described method, the method furtherincludes changing the effective capacity for at least one stage, whereinthe cost of the change in the effective capacity is a marginal cost. Thechange to the net economic value may then be compared with the marginalcost. As such, the improvements may be identified using an automatedprocess of optimization using as a goal the utility combined with flowof candidates and the marginal cost.

At least one additional node may be added to the probabilistic networkwherein a state of the at least one additional node represents a projector a therapy area. The links from the at least one additional node areused to represent systematic differences in criteria between projects orbetween therapeutic areas. A plurality of states of the at least oneadditional node may be selected wherein the improvements the R&D processare evaluated over at least a portion of the R&D organization. Theeffectiveness of a research process may be represented in terms of adiscrete number of candidates in unit time using a Poisson functionapplied to an average flow of candidates. Still yet, the additional nodemay be a hidden node, as described above, or otherwise represent acombination of two or more criteria.

In another variation of the method, the effective capacity may becalculated based on the available resources, wherein the availableresources are movable between stages of the R&D process. As each of thestages includes one or more sets of parallel tasks, estimating theeffective capacity involves calculations based on the availableresources wherein each available resource is either limited to usewithin a stage, a set of tasks or an individual task or is sharedbetween a plurality of tasks.

The conditional and local probabilities used within the method may beobtained through a training process, using a trainable network as theprobabilistic network.

Calculation of the output flow of the candidates involves cumulatingratios of the output flow to the input flow for the sets of paralleltasks. Each ratio may be calculated in one of two ways depending on thenature of the probabilistic network. For a network with ordered discretestates, the ratios are calculated by summing over local probabilities.For networks in which the nodes take continuous values, the ratios arecalculated by integrating over the local probabilities. One or morepairs of thresholds may be used to select research options where eachpair defines a range.

Calculation of the output flow may further include cumulating flowratios using one or more of the group consisting of independentattrition, conversion factors, the number of cycles and the batch size.With such calculation of output flow, the method may further involvecreating one or more additional nodes in the probabilistic networkrepresenting the number of cycles or batch size with one or moreadditional links, wherein each link influences quality attributes,wherein the value of the one or more additional nodes is used incalculating the flow ratios.

The method may further be used to identify the improvements using anautomated process of optimization using as a goal the utility combinedwith flow of candidates. The improvements may also be evaluated using ameasure of risk calculated over the probability distributionsrepresented by the nodes representing quality attributes of thecandidates.

In the context of the inventive method, the probabilistic network mayalso be used to conduct a sensitivity and risk analysis of proposals forinvestment in research technology.

The method can be adjusted to provide any appropriate outcome, anddisplay that outcome or outcomes, such as for example, chemical andbiological data in conjunction with decision boundaries for criteria.

The net value of the various outcomes derived from the subject methods,taking account of end values through a belief network and of capacityloading, could be combined to help management and scientists determinethe best weighting of the various criteria for decision making involvingmultiple attributes where the attributes are numerically combined intonew variables, and decide whether certain tasks and/or test should beperformed in parallel or in series and if so, at what threshold levelsfor one or more of the combined variables.

The subject methods may further include means for factoring in errorsand uncertainties in the scientific experimentation and computationalchemical and biological predictions that are made concerning theproduct, estimated from examples of data. One way in which this can beaccomplished is by incorporating a non-informative prior distribution,for example, as described in “A catalog of noninformative priors”, Yangand Berger (1998) (seehttp://www.isds.duke.edu/˜berger/papers/catalog.html).

The present invention further includes systems for improving theeffectiveness of a R&D process within an organization as describedabove. In one variation, an inventive system includes the following:means for identifying quality attributes of the candidates; means fordefining a measure of utility over a quality profile of identifiedquality attributes, wherein a combination of utility and the flow ofcandidates quantifies improvements to the R&D process; means foridentifying the types of research options for each of the plurality ofstages; means for identifying criteria relevant for use in selectingand/or enumerating amongst research options of each identified type;means for identifying causal relationships amongst the criteria andbetween the criteria and the quality attributes; means for forming aprobabilistic network representing the causal relationships as linksholding conditional probabilities and representing the criteria andquality attributes as nodes, each node having a plurality of statespermitting an ordering amongst the research options consistent withthresholds applicable for selecting and/or enumerating the researchoptions, and each node holding a probability distribution over theplurality of states; means for estimating an effective capacity for eachstage; means for setting local probabilities at nodes that representprior knowledge of the research options and the R&D process; means foridentifying alternatives for operating the R&D process; means forintervening in the probabilistic network to change the probabilities ofthe states to represent differences between the alternatives wherein thedifferences comprise at least one of (i) the criteria to be applied,(ii) the stage in which a criterion is applied, and (iii) the thresholdat which a criterion is applied; and means for evaluating the degree ofimprovement by calculating (i) the quality profile of the candidates,(ii) the change in an average value of the utility for each candidate,and (iii) the output flow of the candidates, limited at each stage bythe input flow relative to the effective capacity.

The system may further include means for visually displaying the variousoutcomes derived from the subject methods. Examples of the outcomes thatmay displayed include but are not limited to the loading on capacity inresponse to user selection of criteria and thresholds, the probabilitiesof states, preferably in the form of a cumulative display of theprobabilities of states within which thresholds are displayed asboundaries between states. The states distinguished by such thresholdsmay be modified by “clicking and dragging” threshold positions, or byclicking on the intended new threshold position. Other outcomes whichmay be displayed include the discrete probability distributions whichmay be portrayed as bars and net values, which may be illustrated in theform of bar charts. Other visual cues, such as change of color of a bar,may be used, for example, where a change leads to a flow at any stageexceeding capacity. The currently preferred range for each criterion maybe highlighted, and the numeric values bounding this range may bedisplayed. Other data that may be displayed includes the name of a statewhere the selection preference is for a state. Still yet, information onthe states may be fed back to the user in the form of an optionaldisplay such as a tool-tip.

The subject systems may further utilize display prompts to prompt theuser to select a criterion having a range that includes, optionally, nostates or all states. The user response may be reflected by a change indisplay such as graying out of the visual representation of thecriterion. The user may also be given the option of disabling the use ofsuch a particular criterion within the calculation of capacity loading.

Input displays that assist the user to make selection betweenalternatives may include pick lists to select potential criteria from awider list of possible criteria for which knowledge is available withina probability network. The location on the display of those criteriathat are to be applied in parallel may be within a spatial arrangementsuch as a stack that implies contiguity. Such stacks representing setsof parallel tasks within a stage may be arranged within boundaries orwithin other visual devices to represent containment. Preferredrelationships between criteria that, a priori, will show a high level ofcorrelation may be shown in the form of families, optionally named (suchas “activity”), and such families may be arranged with a visiblerelationship between tasks, such as a row or column. In one particularembodiment of the subject system, a decision boundary is visuallyoverlaid on a display of specific research options using the values ofthe measures for those options to determine position of representationof the options on the display in such a way that options for which achoice is indifferent will be displayed in the vicinity of the decisionboundary. The position of the decision boundary may be guided ordetermined by a process of optimization over different means forresearch conduct using the estimates of utility and of economic valuedescribed previously,

The systems of the present invention include a computer readable mediumcarrying one or more software programs each having one or more sequencesof instructions for carrying out one or more of the method steps of thepresent invention. The software program may include means for receivingone or more sequences of instructions from a user of a computer systemfor initiating operation of such software program and for providingvarious parameters for performing the method steps. The subject systemsfurther include a processor for storing the one or more softwareprograms, means for executing the software programs and means forstoring the data generated from execution of the software programs.

While the present invention has been described with reference to thespecific embodiments thereof, it should be understood by those skilledin the art that various changes may be made and equivalents may besubstituted without departing from the true spirit and scope of theinvention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processstep or steps, to the objective, spirit and scope of the presentinvention. All such modifications are intended to be within the scope ofthe claims appended hereto.

1. A method of improving the effectiveness of a research and development(R&D) process within an organization comprising one or more sites,research areas and projects, the process comprising a plurality ofstages of selection and/or enumeration of research options, wherein eachstage comprises work comprising one or more sets of parallel tasks whichare optionally reiterated in a number of cycles, wherein each taskcomprises one or more subtasks, wherein a batch is a collection of theone or more subtasks, wherein each set of parallel tasks has an inputflow and an output flow of research options, wherein, at a final stage,the research options are candidates, the method comprising the steps of:a. identifying quality attributes of the candidates; b. defining ameasure of utility over a quality profile of identified qualityattributes, wherein a combination of utility and the flow of candidatesquantifies improvements to the R&D process; c. identifying the types ofresearch options for each of the plurality of stages; d. identifyingcriteria relevant for use in selecting and/or enumerating amongstresearch options of each identified type; e. identifying causalrelationships amongst the criteria and between the criteria and thequality attributes; f. forming a probabilistic network representing thecausal relationships as links holding conditional probabilities andrepresenting the criteria and quality attributes as nodes, each nodehaving a plurality of states permitting an ordering amongst the researchoptions consistent with thresholds applicable for selecting and/orenumerating the research options, and each node holding a probabilitydistribution over the plurality of states; g. estimating an effectivecapacity for each stage; h. setting local probabilities at nodes thatrepresent prior knowledge of the research options and the R&D process;i. identifying alternatives for operating the R&D process; j.intervening in the probabilistic network to change the probabilities ofthe states to represent differences between the alternatives wherein thedifferences comprise at least one of (i) the criteria to be applied,(ii) the stage in which a criterion is applied, and (iii) the thresholdat which a criterion is applied; and k. evaluating the degree ofimprovement by calculating (i) the quality profile of the candidates,(ii) the change in an average value of the utility for each candidate,and (iii) the output flow of the candidates, limited by the input flowof research options relative to the effective capacity at each stage. 2.The method of claim 1, wherein the utility is a measure of grosseconomic value.
 3. The method of claim 2, wherein a net economic valueis the gross economic value less operating costs estimated from at leastthe input flow into at least one stage.
 4. The method of claim 1,wherein the effective capacity has a cost.
 5. The method of claim 4,wherein the cost is a marginal cost.
 6. The method of claim 3, whereinthe effective capacity has a cost and further comprising comparing thenet economic value with the cost of the effective capacity.
 7. Themethod of claim 1, further comprising the step of changing the effectivecapacity for at least one stage.
 8. The method of claim 7, wherein thecost of the change in the effective capacity is a marginal cost and themethod further comprising the step of comparing the change in neteconomic value with the marginal cost.
 9. The method of claim 1, furthercomprising the step of adding at least one node to the probabilisticnetwork.
 10. The method of claim 9, wherein a state of the at least oneadditional node represents a project or a research area.
 11. The methodof claim 10, wherein the links from the at least one additional node areused to represent systematic differences in criteria between projects orbetween therapeutic areas.
 12. The method of claim 10, furthercomprising the step of selecting a plurality of states of the at leastone additional node wherein the improvements are evaluated over at leasta portion of the R&D organization.
 13. The method of claim 1 or 12further comprising the step of representing the effectiveness of aresearch process in terms of a discrete number of candidates in unittime using a Poisson function applied to an average flow of candidates.14. The method of claim 1, further comprising calculating the effectivecapacity based on available resources.
 15. The method of claim 14,wherein the available resources are movable between stages.
 16. Themethod of claim 1, wherein each of the stages comprises one or more setsof parallel tasks, and wherein the step of estimating the effectivecapacity comprises calculations based on available resources whereineach available resource is limited to use within a stage, set or task.17. The method of claim 1, wherein each of the stages comprises one ormore sets of parallel tasks, the method further comprising the step ofcalculating the effective capacity based on the available resourceswherein each available resource is shared between a plurality of tasks.18. The method of claim 1, further comprising adding at least one nodeto the probabilistic network, wherein the at least one additional nodeis a hidden node.
 19. The method of claim 1, wherein at least one noderepresents a combination of two or more criteria.
 20. The method ofclaim 1, where the probabilistic network is a trainable network, furthercomprising the step of obtaining the conditional and local probabilitiesthrough a training process.
 21. The method of claim 1, farthercomprising evaluating the improvements using a measure of riskcalculated over the probability distributions represented by the nodesrepresenting quality attributes of the candidates.
 22. The method ofclaim 1, wherein calculating the output flow of the candidates comprisescumulating ratios of the output flow to the input flow for the sets ofparallel tasks and including the conversion factors between differingresearch options at different stages.
 23. The method of claim 22,wherein the probabilistic network comprises ordered discrete states, andeach ratio is calculated by summing over local probabilities.
 24. Themethod of claim 22, wherein the probabilistic network comprises nodeswhich take continuous values, and wherein each ratio is calculated byintegrating over local probabilities.
 25. The method of claim 22,wherein calculating the output flow further comprises cumulating flowratios using one or more of the group consisting of independentattrition, unit changes, the number of cycles and the batch size. 26.The method of claim 25, further comprising the step of creating one ormore additional nodes in the probabilistic network representing thenumber of cycles or batch size with one or more additional links,wherein each link influences quality attributes, wherein the value ofthe one or more additional nodes is used in calculating the flow ratios.27. The method of claim 1, further comprising identifying and comparingthe possible improvements using an automated process of optimizationusing as a goal the utility combined with flow of candidates.
 28. Themethod of claim 1, further comprising the step of using theprobabilistic network to conduct a sensitivity and risk analysis ofproposals for investment in research technology.
 29. The method of claim1, further comprising the step of displaying chemical and biologicaldata in conjunction with decision boundaries for criteria.
 30. Themethod of claim 8, further comprising identifying and comparing thepossible improvements using an automated process of optimization usingas a goal the utility combined with flow of candidates and the marginalcost.
 31. The method of claim 1, further comprising reiterating steps hthrough k.
 32. The method of claim 1, wherein the thresholds compriseone or more pairs of thresholds, wherein each threshold pair defines arange for selecting the research options, the method further comprisingusing the threshold pairs to select research options.
 33. The method ofclaim 1, further comprising comparing the flow through a parallel set oftasks to the amount of work required for one or more tasks within aproject and using the flow to estimate the time required to complete theset.
 34. A system for improving the effectiveness of a research anddevelopment (R&D) process within an organization comprising one or moresites, research areas and projects, the process comprising a pluralityof stages of selection and/or enumeration of research options, whereineach stage comprises work comprising one or more sets of tasks which areoptionally reiterated in a number of cycles, wherein each task comprisesone or more subtasks, wherein a batch is a collection of the one or moresubtasks, wherein each set of parallel tasks has an input flow and anoutput flow of research options, wherein, at a final stage, the researchoptions are candidates, the system comprising: a. means for identifyingquality attributes of the candidates; b. means for defining a measure ofutility over a quality profile of identified quality attributes, whereina combination of utility and the flow of candidates quantifiesimprovements to the R&D process; c. means for identifying the types ofresearch options for each of the plurality of stages; d. means foridentifying criteria relevant for use in selecting and/or enumeratingamongst research options of each identified type; e. means foridentifying causal relationships amongst the criteria and between thecriteria and the quality attributes; f. means for forming aprobabilistic network representing the causal relationships as linksholding conditional probabilities and representing the criteria andquality attributes as nodes, each node having a plurality of statespermitting an ordering amongst the research options consistent withthresholds applicable for selecting and/or enumerating the researchoptions, and each node holding a probability distribution over theplurality of states; g. means for estimating an effective capacity foreach stage; h. means for setting local probabilities at nodes thatrepresent prior knowledge of the research options and the R&D process;i. means for identifying alternatives for operating the R&D process; j.means for intervening in the probabilistic network to change theprobabilities of the states to represent differences between thealternatives wherein the differences comprise at least one of (i) thecriteria to be applied, (ii) the stage in which a criterion is applied,and (iii) the threshold at which a criterion is applied; and k. meansfor evaluating the degree of improvements by calculating (i) the qualityprofile of the candidates, (ii) the change in an average value of theutility for each candidate, and (iii) the output flow of the candidates,limited at each stage by the input flow relative to the effectivecapacity.
 35. A computer readable medium carrying one or more sequencesof instructions from a user of a computer system for improving theeffectiveness of a research and development (R&D) process within anorganization comprising one or more sites, research areas and projects,the process comprising a plurality of stages of selection and/orenumeration of research options, wherein each stage comprises workcomprising one or more sets of tasks which are optionally reiterated ina number of cycles, wherein each task comprises one or more subtasks,wherein a batch is a collection of the one or more subtasks, whereineach set of parallel tasks has an input flow and an output flow ofresearch options, wherein, at a final stage, the research options arecandidates, wherein the execution of the one or more sequences ofinstructions by one or more processors causes the one or more processorsto perform the steps of: a. identifying quality attributes of thecandidates; b. defining a measure of utility over a quality profile ofidentified quality attributes, wherein a combination of utility and theflow of candidates quantifies improvements to the R&D process; c.identifying the types of research options for each of the plurality ofstages; d. identifying criteria relevant for use in selecting and/orenumerating amongst research options of each identified type; e.identifying causal relationships amongst the criteria and between thecriteria and the quality attributes; f. forming a probabilistic networkrepresenting the causal relationships as links holding conditionalprobabilities and representing the criteria and quality attributes asnodes, each node having a plurality of states permitting an orderingamongst the research options consistent with thresholds applicable forselecting and/or enumerating the research options, and each node holdinga probability distribution over the plurality of states; g. estimatingan effective capacity for each stage; h. setting local probabilities atnodes that represent prior knowledge of the research options and the R&Dprocess; i. identifying alternatives for operating the R&D process; j.intervening in the probabilistic network to change the probabilities ofthe states to represent differences between the alternatives wherein thedifferences comprise at least one of (i) the criteria to be applied,(ii) the stage in which a criterion is applied, and (iii) the thresholdat which a criterion is applied; and k. evaluating the degree ofimprovements by calculating (i) the quality profile of the candidates,(ii) the change in an average value of the utility for each candidate,and (iii) the output flow of the candidates, limited at each stage bythe input flow relative to the effective capacity.